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ABSTRACT 

The specific focus of this research was to determine 
how the outcomes on two alternative forms of assessment 
(multiple-choice and hands-on/manipulative) for science process 
skills were related when students were grouped on the basis of sex, 
race/ethnicity, and poverty level. Subjects were 1,381 fourth graders 
in a culturally diverse city school district in New York state. Also 
explored was the relationship between the reading scores of students 
and their scores on the two alternative science assessments and the 
effect of the presence of an elementary science specialist on the 
student scores. The reliability of New York's Elementary Science 
Program Evaluation Test (ESPET) was also considered. All students 
performed better on the hands-on test, and the traditional gap 
associated with socioeconomic or racial/ethnic status was reduced by 
the hands-on test. Economic status appeared to have an effect on 
science achievement, and reading had a greater effect than poverty on 
science achievement. Results supported the reliability of the ESPET, 
but did not support the importance of an elementary science 
specialist to student achievement. Results show that all students can 
learn science given appropriate instruction. Thirteen tables and 17 
figures present st udy findings. (Contains 48 references.) (SLD) 
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Allernalive Forms of Assessment in Elementary Science: The Interactive Effects of Sex, Reading, 
Race, Economic Level and the Elementary Science Specialist on Hands-on and Multiple-Choice 

Assessment of Science Process Skills 

Introduction* 

The publication of The Bell Curve (Hernstcin & Murray, 1994), currently on the "best- 
seller" list, overtly contradicts a statement that many educators frequently hear and say: "All 
children can learn." The results of the research reported in this paper provide a counterargument 
for their stance. Regardless of economic level, gender, race/ethnicity, or reading level, this 
research pro\'idcs evidence that aH children can learn science given two prerequisites. First, to be 
certain that the science program is taught, designate specific teachers as 'science specialists' to 
deliver instruction, elementary teachers who want to teach science, who are enthusiastic about 
teaching science, who will give the time and energy to the teaching of science, and who wall be 
accountable fordoing so. This will provide the appropriate conditions for all children to have the 
opportunity to learn science. Second, in addition to traditional multiple-choice testing of science, 
provide an alternative, hands-on, performanced-based assessment. This will pro\'idc students with 
more than one way to demonstrate what they know and can do. Based upon the results of this 
research, it is clear that we need to broaden our concepts regarding how and by whom instruction 
is delivered and how we assess. 

The research described in this paper was carried out in an attempt to resoh e some ol' the 
descrepancies and confront some of the issues with which many science educators continue to 
struggle regarding assessment. Howe\'er, when we assess and hold students responsible for what 
they have learned, we arc also socially and intellectually obligated to address the assessment of 
their opportunity to learn the content and or skills which arc being assessed. Educators must find 
and employ ways to measure the existence and the quality of the resources for teaching and 
learning science, as well as to identify and use alternative methods which allow students to better 
demonstrate what they know and can do. The literature rc\'cals that some research has been done 
on the latter in the form of comparing performance on hands-on \'crsus multiple-choice papcr-and- 
pcncil forms of assessment (Shavelson, Baxter, & Pine 1992; Doran, 1990; Doran & Tamir, 1992; 
Kuechle, 199(); Comber & Kcc\'cs, 1973). Other researchers (Doran & Tamir 1992; Kucchlc, 
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1990; Comber & Keeves, 1973) have reported that students were obser\'ed to perform better on 
hands-on tests than on multiple-choice tests of science process skills. The research reported in this 
paper supports and goes beyond their work. 

Using the New York State Elementar)' Science Program Evaluation Test (ESPET), the test 
score gap on the multiple-choice test that is observed between students of different economic, racial 
and ethnic backgrounds was found to be reduced on the hands-on test; and the hands-on test 
format section of the ESPET was able to discriminate between two groups of students, those who 
had had a science program with an elementary science specialist and those students who had not 
had a program with an elementary science specialist. Thus, it was observed that, when provided 
with the opportunity to learn, the test score gap between subgroups was reduced on the hands-on 
test. Students in science programs with a science specialist performed significantly better that those 
in programs without a science specialist, especially those students of racial/ethnic groups currently 
underrepresented in the sciences and those with low-reading and high po\'erty levels. When 
students were provided a science specialist, the opportunity to learn science increased for all 
subgroups of students, but because some subgroups exhibited a greater increase in scores with a 
science specialist than other subgroups, a decrease in the test score gap between subgroups was 
observed. The multiple-choice test of science prcx:ess skills was not able to discriminate between 
the two groups of students (those provided with a science specialist and those without a science 
specialist) and the multiple-choice test o^' science prcx:ess skills in no way indicated the impact oi 
the science specialist on the various subgroups of students. Under the twin conditions of a science 
specialist (which provides the opportunity to learn science) and hands-on assessment techniques 
(which provide students the opportunity to demonstrate what they ha\'e learned), we observe that 
all students can learn science. 
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Alternative Forms of Assessment in Elementary Science: The Interactive Effects of Sex, Reading, 
Race, Economic Level and the Elementary Science Specialist on Hands-on and Multiple-Choice 

Assessment of Science Process Skills 

Objectives. 

The specific focus of this research was to determine, for fourth graders in a culturally diverse 
city school district in New York State, how the outcomes on two alternati\'e forms of assessment 
(multiple-choice and hands-on/manipulative) for science process skills were related when students 
were grouped on the basis of sex, race/ethnicity, and poverty level. In addition, two subproblems 
were also explored: the. relationship between the reading scores of students and their scores on the 
two alternative forms of science assessment; and the effect of the presence of an elementary science 
specialist on the scores attained by students on the two alternative forms of assessment. The 
research addresses twelve questions: Did the NYS ESPET pro\'e to be a reliable test? What was 
the relationship between the multiple-choice and hands-on science process skill test scores? Did 
the measurement of science process skills vary with the method of assessment used? Was the test 
score gap between student subgroups reduced when students were tested in an alternati\'e way? 
Were there differences in performance based upon race? Were there differences in performance 
based upon po\'erty level? Were there differences in performance based upon sex? What was the 
relationship between students' reading scores and their multiple-choice and hands-on science 
scores? Was test performance more affected by race or poverty? Was test performance more 
affected by poverty or reading? How did the presence of an elcmentar>' science specialist affect 
hands-on and multiple-choice scores? Docs alternative assessment make a difference? 

Conceptual Framework and Relationship to the Literature. 

Educational theorists prof>osc that assessment should match pedagogy. Hands-on tests of 
science process skills may be considered as coming closer to measuring what science educators 
want to measure (Doran, 1990; Kaniji, 1988; Cizck, 1991; Pctraitis, 1991; Kulm and Stcussy, 
1991, Shavclson, Baxter, and Pine, 1992, Mcng & Doran, 1990; Mitchell, 1992a; Wiggins, 1989, 
1990), i.e., the science process skills. The four-year sludy by Shavclson, Baxter, and Pine (1992) 
provides some of the first evidence that hands-on assessments measure aspects of science 
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achievment that arc different from those measured by multiple-choice tests. The First International 
Science Study (Comber and Kceves, 1973) and the Second International Science Study (Doran and 
Tamir, prepublication manuscript, 1992) reveal some of the first evidence that elementary students 
score higher on hands-on tests than on multiple-choice tests of science process skills. Keuchlc 
(1990) and Marshall (1991) report similar results. SaUler (1988) reports that different subgroups 
of children (American Indian and Hispanic-American children) obtain higher scores on 
performance IQ tests than on verbal IQ tests. Hein ( 1987) conveys that most science testing at the 
elementary level conflicts with objectives and program emphasis. The teaching of the science 
process skills through hands-on science experiences should be assessed using hands-on, 
performance tasks and not by multiple-choice, paper and pencil tests. Pine (1990), Davis & 
Armstrong (1991) and Champagne (1990) support Hein and stress the close connection between 
assessment and pedagogy and the need to articulate how a particular form of assessment matches 
pedagogic beliefs. According to Champagne (1990), the closer the assessment task is to what one 
wants to assess, the closer the sores will be to a true measure of attainment of the skill or concept. 

Wadsworth's ( 1984) explanation of Piaget's theory of cogniti\'e development may be applied 
to models for assessing science prcx:ess skills: the concrete operational child (ages 7-1 1) can use 
logical operations to solve only problems that involve concrete objects and events in the immediate 
present. Hands-on tests are at the concrete level; multiple-choice tests are at the abstract level. 
Students should be taught and tested at their cognitive level of development. The students in this 
study were fourth graders, most of whom were age nine and most likely in the middle of the 
concrete operational lc\'el. 

Schcx^l variables have also been shown to affect science achievment more than other subjects 
such as reading (Tamir, 198*'^). Zu/ovsky and Tamir ( 1989) examined the relative status of school 
(alterable) variables versus home (fixed) variables and found that the contribution of school 
variables was bcnh subject specific and system specific, e.g., school variables had more of an 
effect on science achievment especially in low scx:ioeconomic schcx^ls. An elementary science 
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specialist is a controllable school variable. Abell (1990, Williams (1990), and Hounshcll and 
Swartz (1987) present their opinions for and against the use of elementary science specialists but 
there is little hard data that compare the science achievment of elementary students wiih and without 
a science specialist. 

The ability of a child to read a science test may influence performance. Differences in 
reading ability exist between students of different economic, racial and ethnic backgrounds (Scott- 
Jones and Clark, 1986; Jones, 1984). Economically disadvantaged students were found to achieve 
more in activity-based science programs than in text-book based science programs (Beane, 1985; 
Bredderman, 1982; Shymansky, Kyle and Alport, 1982). Research of the literature revealed that 
limited information was available on how race, sex, and economic status might interact to affect the 
science achievement of blacks, females, and disadvantaged students (Oakes, 1990). Few 
researchers have been able to focus on both race and sex, and they have frequently confounded 
economic status with minority group membership. In the research reported here, test score data for 
students for whom all these characteristics were known and were available for analysis. 

Methods and Data Sources, 

This was a quasi-experimental study of 1381 fourth grade students in a city school district in 
New York State. (Table 1 1 describes the students in this study.) All fourth grade students in New 
York State take all three parts of the New York State Elementary Science Program Evaluation Test 
(ESPET) which contains two methods for assessment of science process skills: hands-on and 
multiple-choice. The ESPET, constructed by the New York State Department of Exlucation and 
mandated to be adr. -nistered each May to all fourth graders, is administered and scored according 
to standardized procedures as described in the document produced by the New York State 
Education Department entitled Program Evaluation Test in Science Grade 4: Directions for 
Administering and Scoring. The main and interactive elTects of three learner attributes (sex, 
racc/cthnicity, and poverty le\'cl) on each method of assessment were examined. Racc/cthnicity 



() 



Alternative Forms of Assessment in Elementary Science: The Interactive Effects of Sex, Reading, 
Race, Economic Level and the Elementary Science Specialist on Hands-on and Multiple-Choice 

Assessment of Science Prcx:ess Skills 



group membership was assigned according to categories used by New York State for all reporting 
purpc^ses: White, Black, Hispanic, Other. Poverty level was established as high poverty, low 
pcwerty, or no poverty according to the Free and Reduced Meal Policies Eligibility Guidelines as 
set forth by the NYS Education Department, Bureau of Food Management and Nutrition. To 
determine the effects of the three independent variables (sex, race/ethnicity, and poverty level) on 
each of the two dependent variables (multiple-choice score and hands-on score), the independent 
variables were organized into a 2 x 3 x 3 factorial design. In addition, the main and interactive 
effects of two other independent variables (reading level and the presence of an elementary science 
specialist) were also explored. Descriptive statistics such as group means and standard deviations 
were computed, various correlation coefficients were determined, and t-tests, analysis of variance, 
and Tukey post hoc procedures were carried out. Reliability coefficients (Cronbach alpha and split 
half) were determined for the test used (the 1989 New York State Elementary Science Program 
Evaluation Test) and for its subparts. 

The 1989 ESPET consists of three parts. Part I has 29 items and assesses science content in 
multiple-choice format. Part II with 16 items uses the multiple-choice test format and assesses 
science process skills. Part III with 15 items also assesses science process skills but uses a hands- 
on format. Part I has a maximum raw score of 29. Part II has a maximum raw score of 16. Part 
III consists of 5 different stations wuth a \'ariety of tasks for a maximum raw score of 22. Raw 
scores were con\'erted and reported as percent correct on each part. Parts II and III both measured 
the process skills listed in the following paragraph except for two skills, creating models and 
replicating, which were not measured on either Part II or Part III. 

Definitions of the prcx:css skills tested by the ESPET are given in the New York State 
Elementary Science Syllabus (1985): classifying, creating models, formulating hypotheses, 
generalizing, identifying variables, inferring, interpreting data, making decisions, manipulating 
materials, measuring (length, mass, volume, and temperature), observing, predicting, recording 
data, replicating, using cues, developing v(x:abulary, and using numbers. The ESPET was 
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developed by committees of science educators (pre-college teachers and university researchers) 
from across New York State, not by commercial test publishers. The consultants wrote, edited, 
and selected the test items under the direction of members of the New York Stale Education 
Department Bureau of Science Education and Bureau of Testing. Manipulative items were 
developed and adapted from a pool of hands-on items from other assessment instruments {the First 
and Second International Association for the Ex'aluation of Educational Achievement Science Study 
(FISS and SISS) and the British Assessment of Performance Unit (APU)}. For the multiple- 
choice questions, writers submitted test items to a test pool. All test items were keyed directly to 
the 1985 New York State Elementary Science Syllabus . Other consultants edited the test questions 
and others selected the items to be field-tested in various schools across New York Slate. Other 
consultants chose the questions from those field-tested items for the final form of the test. This test 
construction process is used by the New York State Department of Education to construct all of its 
statewide tests and is assumed lo produce an instrument of high content and construct validity. 

Results , 

The findings presented in the sections that follow are from a study conducted in a city schcx:)! 
district in New York State where the ESPET scores of 1381 foi rth graders were analy/xd 
(Saturnelli, 1993). 

Did the ESPET prove to be a reliable test? 

A Cronbach Alpha of .87 was obtained for the entire ESPET (Parts I, II and III combined). 
For Part I, science content using multiple-choice format, a split-half reliability coefficient of .75 
was obtained. For Part II, science process skills using multiple-choice format, a split-half 
reliability ccxifficicnt of .78 was obtained. For Part III, science process skills using hands-on 
format, a reliability ccx^fficicnt of .72 (Cronbach's Coefficient Alpha) was obtained. The lower 
\ alucs for the subparts can be attributed to the smaller number of items on each part as compared to 
the number of items on the total test. In addition, the correlation between the total ESPET and the 
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Science Subtest of the Iowa Tests of Basic Skills (ITBS) was found to be .69, a mcxierately strong 
correlation according to Guilford (1956). This data gives support for the criterion related 
(concurrent) validity of the ESPET. Group difference data gives suppcut for the construct validity 
of the hands-on test. Students who had a science program which included an elementary science 
specialist once ever>' six days for instruction (N = 577) had significantly higher scores (t = 9.52, 
p = 0.001) on the hands-on science prcx:css skills section of the ESPET than those students who 
did not have a science specialist (N = 804). However, the presence of a science specialist did not 
significantly affect the scores of students on the multiple-choice science process skills section. The 
multiple-choice science process skills test was not able to discriminate between those students who 
did and those who did not have a science specialist but the hands-on science process skills test did 
detect differences between these two groups. 

What Is the Relationship Between the Multiple-Choice and Hands-On Scores? 

For the total sample of students in the study, a .51 correlation coefficient (PPMC) between 
the two tests of science process skills was obtained (Table 1). However, analysis of data 
disaggregated on the basis of race reveals interesting relationships otherwise not detected. For 
Hispanics, the relationship between their hand-on and the multiple-choice scores was higher than it 
was for the toal sample (r = .60) while for Blacks, the relationship was lower than it was for the 
total sample (r = .36). This higher correlation for Hispanic students and lower correlation for 
Black students between the hands-on and multiple-choice science skills scores must be fuithcr 
examined with the following information about reading scores (Table 2). For Hispanic students, 
the correlation between reading scores and their science prcx:ess skills test scores is .59 (hands-on) 
and .70 (multiple-choice). For Black students, the correlation between reading scores and their 
science process skills test scores is .29 (hands-on) and .52 (multiple-choice). Hispanic students 
appeared to be more dependent upon reading for bc^th the hands-on and multiple-choice tests and 
therefore performed about the same on both Ibrms whereas the Black students, apparcntlx' less 
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dependent upon reading on the hands-on test than on the multiple-choice test, performed better on 
the hands-on form than on the multiple-choice form. Because reading was found to be associated 
more with the multiple-choice test than with the hands-on test (Table 2), partial correlations were 
run controlling for reading. A partial correlation coefficient of .30 was obtained between the two 
tests of science process skills. These differences are discussed in further detail in the section on 
reading which follows. The correlations with the ITBS science subtest discussed in the preceeding 
section and the correlations with the ITBS reading scores discussed in this section indicate that, 
while measuring some of the same aspects of science achievment, the two tests of science process 
skills may very well be measuring some different aspects of science achievement. 



Table 1. Relalionship Between Multiple Choice (MC) and Hands-on (HQ) Science Process Skills Test Scores 
tor Varipus Student Subgroups 
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Table 2 Correlations: ITBS Reading Scores with Science Process Skills Test Scores bN' Sex and Rac e 
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Does the Measurement of Science Process Skills Vary With the Method of 

Assessment? 

Studcnls of every subgroup except Others were found to score significantly higher on the 
hands-o test of science process skills than the}' did on the multiple-choice test of science prtKX-ss 
skills. In nearly every case, the difference was significant (Table 3). 

Table 3 X'ariation in the Measurement of Science Process Skills for Student Subgroups 
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Was the Test Score Gap Between Student Sub£roups Reduced 
When Students Were Tested in an Alternative Way? 

The test score gap between various subgroups was reduced when students were tested using 
the hands-on formal. The greatest reduction in test score gap was observed between the following 
groups: no-poverty White males and high-poverty White males; no-poverty Whites and high- 
poverty Whites; no-poverty White males and no-poverty Black males; no-poverty White females 
and nopoverty Hispanic females; White males and Black males; White females and Black females 
(Table 4). 

Table 4 Test Score Gap Between Various Student Subgroups 



Groups being compared N MC Test HO Test Gap Reduced 

Score Gap Score Gap by 



White males no-poverty and 


334 








high-poverty 


41 


13 


6 


7 


Whites no-poverty and 


652 








high- poverty 


77 


12 


5 


7 


No-poverty females White and 


322 








Hispanic 


51 


10 


3 


5 


No-poverty males White and 


334 








Black 


74 


14 


10 


4 


Males White and 


383 








Black 


174 


10 


12 


4 


Females White and 


375 








Black 


194 


10 


12 


4 


Whites and 


758 








Blacks 


368 


16 


13 


3 


Males no- poverty and 


457 








high-px)verty 


174 


13 


10 


3 


White females no-pov(Tty and 


322 








high-poverty 


37 


9 


0 


3 


Black females no- poverty and 


62 








high-poverty 


112 


7 


4 


3 


No- poverty and 


899 








high-poverty 


388 


14 


12 


2 


l^emales no- poverty and 


444 








high px)verty 


212 


15 


13 


2 


No-poverty females White and 


322 








Black 


02 


12 


10 


2 


High-poverty females White and 37 








Black 


112 


10 


8 


2 


Females White and 


375 








Hispanic 


123 


14 


12 


2 


Whites and 


758 








Hispanics 


111 


15 


14 


1 


Blacks no- poverty and 


137 








high-p<A'erty 


200 


0 


5 


1 


Males White and 


383 








Hispanic 


104 


10 


10 


0 


Black males no-povert) i\nd 


74 








high- poverty 


104 


4 


4 


0 
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Alternative Forms of Assessment in Elementaiy Science: The Interactive EfTects of Sex, Reading, 
Race, Economic Level and the Elementary Science Specialist on Hands-on and Multiple-Choice 

Assessment of Science Process Skills 

Were There Differences in Performance Based upon Race? 

When data was not disaggregated. Whites were observed to score significantly' higher than 
Blacks and Hispanics on both the hands-on and multiple-choice tests of science process skills, and 
no significant difference was observed between Blacks and Hispanics. However, when data were 
disaggregated on the basis of sex, race, and poverty level, the following differences between Black 
and Hispanic racial subgroups were observed. For high-poverty groups, on the hands-on test, 
high-poverty Blacks scored significantly higher than high-poverty Hispanics (t = 2.24, p = .026) 
and high-poverty Black females scored significantly higher than high-po\'erty Hispanic females 
(t = 2. 14, p=.034). Additional obser\'ations about the performance of racial groups were made on 
disaggregated data. Detailed discussion follows in the paragraphs below. 

Were There Differences in Performance Based upon Poverty Level? 

On bc^th the hands-on and multiple-choice tests, no-poverty groups scored significantly 
higher than low- and high-poverty groups and no significant difference was obser\^ed between 
low- and high-poverty groups (Tukey, p< .05). For each racial group, as poverty level decreased 
the mean scores increased on both the hands-on and multiple-choice tests (Table 5, Figures 1 and 
2). Po\'erty le\'el appeared to have the greatest effect on the performance of Hispanic students on 
the hands-on test where the mean score of the no-poverty Hispanic students was observed to be 
8.4 points higher than the performance of the high-pcwerty Hispanic students. Poverty level 
appeared to have affected the performance of White students more on the multiple-choice than on 
the hands-on test. On the multiple-choice test, the difference between the mean scores of high- 
pcwerty White students and no-pcwerty White students on the multiple -choice test was obser\'ed to 
be 1 1.3 points, whereas on the hands-on test the difference in mean scores between high-poverty 
and no-poverty Whites was only 6. 1 points. This means that the hands-on test closed the gap 
between high-poverty and no-poverty White students by a factor of 1.9. 
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'I able 5 Moan Scores by Race & Poverty Level on Hands-on and Multiple-Choic e lesls o t 
Science Process Skills 



Mean Scores 
(N) 



Hands-on Mulliple-Choice 





White 


Black 


Hispanic 


While 


Black 


Hispanic 


combined poverty 


78.7 


66.4 


64.7 


70.6 


54.8 


55.7 


levels 


(759) 


(368) 


(226) 


(759) 


(368) 


(226) 


High- 


73.4 


65.1 


60.1 


60.8 


52.8 


53.1 


poverty 


(77) 


(201) 


(107) 


(77) 


(201) 


(107) 


Low- 


74.5 


61.8 


70.2 


62.6 


52.5 


56.2 


poverty 


(30) 


(30) 


(29) 


(30) 


(30) 


(29) 


No 


79.5 


69.4 


68.5 


72.1 


. 58.2 


58.5 


povert>' 


(052) 


(137) 


(90) 


(652) 


(137) 


(90) 




White Black Hispanic 
Racial Subgroup 



Figure 1 . Effects of Poverty Level on Hands-on Test of Science Prcx:ess Skills for Different 
Racial Subgroups 




Allernalive Forms of Assessment in Elementary Science: The Interactive Effects of Sex, Reading, 
Race, Economic Level and the Elementary Science Specialist on Hands-on and Multiple-Choice 

Assessment of Science Prcx:ess Skills 



80 -/r 




White Black Hispanic 
Racial Subgroup 

Figure 2. Effects of Poverty Level on Multiple Choice Test of Science Prcxess Skills for 
Different Racial Subgroups 

Were There Differences in Performance Based Upon Sex? 

No significant differences were observed between the scores ot males and Icmales on either 
the hands-on test or the multiple-choice test until data disaggregated on the basis of sex, race and 
poverty level were analyzed. The following results were then observed. On the hands-on test, no- 
poverty Hispanic females scored significantly higher than no-poverty Hispanic males (t = 2.59, 
p = .011). On the multiple-choice test, no-poverty Hispanic females also scored higher that no- 
poverty Hispanic males but the difference was not significant at the .05 level (t = 1.8(),p = .075). 
Other differences were observed between males and females on the multiple-choice test, where for 
no-poverty students without a science specialist, females scored significantly higher than males (t = 
L97, p =.049). With a science specialist, however, this gap between no-poverty males and 
females did not exist. 
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What Is the Relat^.onship Between Students' Reading Scores and Their Scores 
on the Multiple-Choice and Hands-On Scores? 

For all students, performance on the hands-on test of science process skills was found to be 
less associated with reading than was performance on the multiple-choice test of science prcxess 
skills. The correlation coefficient for reading with the hands-on test was found to be .47 while the 
correlatio! I coeff icient for reading with the multiple-choice test was .64 (Table 2). 

Scores of Hispanic students appeared to be more associated with reading than the scores of 
Black and White students. Both the hands-on and multiple-choice scores of high-poverty students 
were found to be less associated with reading than were the scores of no-poverty students (Table 2). 

A complex pattern of differential performance in reading was observed. In middle- and 
low-reading groups, significant differences (p < .05) in reading scores were found to exist 
between males and females in favor of females. However in the high-reading group, a significant 
difference (p < .01) in reading scores was also found to exist but in favor of males (Table 6, 
Figure 3). 

Table 0 1-tests: Differences in Science Test Performance and Reading Level Between Student Subgroups 
Categorized by Sex and Reading 



Sub- 
group 

N 


Hands-on Skills 
t- 

moan value 


2-taiI 
prob 


Multiple-Choice Skills 
t- 2-taiI 
mean value prob 


Reading ITBS 
t- 2-taiI 
mean value prob 


High-reading 
Male 208 
Female 257 


83.9 
81.7 


L77 


.078 


81.4 
78.7 


2.00 


.046 


76.6 
74.6 


2.64 


.009 


Middle- reading 
Male 212 
Female 251 


73.8 
72.9 


.61 


.540 


64.2 
61.6 


1.73 


.084 


55.1 
56.0 


-2.11 


.035 


Low- reading 
Male 257 
Female 197 


05.4 
60.6 


2.51 


.013 


49.8 
48.5 


.71 


.478 


36.6 
38.1 


-1.97 


.049 



Alternative Forms of Assessment in Elementary Science: The Interacti\'e EiTects of Sex, Reading, 
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Assessment of Science Process Skills 




male 
female 



low middle high 

reading reading reading 

Reading Level 

Figure 3. Differences in Performance of Males and Females of Different Reading Le\ els on the 
low'a Tests of Basic Skills Reading Subtest 

On the hands-on test of science process skills, for every reading level, males scored higher 

than females but the only significant difference was at the low-reading level (t = 2.51, p = .013) 

(Table 6, Figure 4). 




Figure 4. DitTcrences in Performance of Males and Females of Different Reading Levels on 
Hands-on Test of Science Prcx^ess Skills 
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Race, Economic Level and the Elementary Science Specialist on Hands-on and Multiple-Choice 

Assessment of Science Process Skills 



On the multiple-choice test of science process skills, for every reading level, males also 
scored higher than females, but the only significant difference was in the high-reading group (t = 
2.00, p = .046) (Table 6, Figure 5). 




male 
female 



lov^ middle high 

reading reading * reading 



Reading Level 



Figure 5. Differences in Performance of Males and Females of Different Reading Lc\'cls on 
Multiple-Choice Test of Science Process Skills 

Significant two-way interactive effects of race and reading were observed for both the 
hands-on and multiple-choice tests. In the low-reading group, Hispanic students scored below 
White and Black students on both the hands-on and multiple-choice tests of science prcx:css skills. 
However, in the middle- and high-reading reading groups, Hispanic students scored above Black 
students but below White students on both the hands-on and multiple-choice tests (Tab' i 7 and 
Figures 6, 7). 

For each racial group as reading scores increased, scores increased on both the hands-on and 
multiple-choice tests of science process skills (Table 7, Figures 8, 9). Examination of the data in 
Table 2 as well as the data in Table 7 reveals that although reading affected the performance of all 
students, it appeared to have the greatest cf feet on Hispanic students. Mean score differences on 
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the hands-on test between low-reading and high-reading Hispanic students was 29.5 points. Mean 
score differences on the multiple-choice test between low-reading and high-reading Hispanic 
students was 35.5 points. 

On the hands-on test, the gap between mean scores of all White and all Hispanic students 
was 14.0 points, whereas the gap between scores of high-reading White and high-reading 
Hispanic students was only 3.2 points. Therefore, the gap between scores of White and Hispanic 
students on the hands-on test is reduced by 77% when reading is removed as a factor. On the 
multiple-choice test, the gap between mean scores of all White and all Hispanic students was 14.9 
points, whereas the gap between scores of high-reading White and high-reading Hispanic students 
was only 53 points. This may be interpreted to mean that the gap between scores of White and 
Hispanic students on the hands-on test is reduced by 65% when reading is removed as a factor. 
Further analysis of the scores on the hands-on and multiple-choice tests for high-reading White and 
high-reading Hispanic students reveals that because there is a 3.2 pcMnt difference between the 
mean scores on the hands-on test and a 5.3 difference between scores on the multiple-choice test, 
by using the hands-on test, the gap between high-readingWhitc and Hispanic students was reduced 
bv 40%. 

On the hands-on test, the gap between the scores of all White and all Black students is 12.3 
points, whereas the gap between the scores of high-reading White and high-rciiding Black students 
is 9.3. This may be interpreted to mean that when reading is not an obstacle, the gap between the 
pertbrmance of White and Black students on the hands-on test is reduced by 24%. On the 
multiple-choice test, the gap between the scores of all White and all Black students is 14.9 points, 
whereas the gap between the scores of high-reading White and high-reading Black students is 9.5. 
This may be intcrrprctcd to mean that when reading is not an obstacle, the gap between the 
performance of White and Black students on the multiple-choice test is reduced by 36%\ 



ERLC 



19 



Alternative Forms of Assessment in Elementary Science: The Interactive Effects of Sex, Reading, 
Race, Economic Level and the Elementary Science Specialist on Hands-on and Multiple-Choice 

Assessment of Science Process Skills 



Table 7 Interactive Effects of Race and Reading on Science Process Skills Tests 



Mean Scores 
Hands-On Multiple-Choice 



White Black Hispanic White Black Hispanic 

(N) (N) (N) (N) (N) (N) 



Low- 


71.5 


61.8 


51.5 


56.4 


46.6 


40.9 


Reading 


(171) 


(180) 


( 98) 


(171) 


(180) 


( 98) 


Middle- 


76.1 


68.7 


71.6 


65.5 


57.4 


61.8 


Reading 


(253) 


(122) 


( 83) 


(253) 


(122) 


( 83) 


High- 


84.2 


74.9 


81.0 


81.7 


72.2 


76.4 


Reading 


(335) 


( 66) 


( 45) 


(335) 


( 66) 


( 45) 


All 


78.7 


66.4 


64.7 


70.6 


54.8 


55.7 


students 


(759) 


(368) 


(226) 


(759) 


(368) 


(226) 



Note: For Hands-On Test, Race by Reading ANOVA F = 10.39'J p = .0001; for 
Multiple-Choice Test, Race by Reading ANOVA F = 6.321, p = .0001. 




Figure 6. Interactive Effects of Race and Reading on Hands-on Test of Science Prcx;ess Skills 
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Low- Middle- High- 

reading reading reading 

Reading Level 



Figure 7. Interactive Effects of Race & Reading on Multiple-Choice Test of Science Process Skills 




Figure 8. Effects of Reading Level on Hands-on Test of Science Process Skills for Different 
Racial Subgroups 
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White Black Hispanic 
Racial Subgroup 



n Low Reading 
Wi Middle Reading 
High Reading 



Figure 9. Effects of Reading Level on Multiple Choice Test of Science Process Skills for Different 
Racial Subgroups 



Was Test Performance Affected More by Race or Poverty? 

Poverty level was observed to have a greater effect than race on both the hands-on and 
multiple-choice science process skills test scores. Across each racial group (White, Black, 
Hispanic) scores were found to increase from high-poverty to no-poverty levels (Table 5). In all 
cases, the gap between racial subgroups was less where scores of students of the same poverty 
levels were compared than when scores of students of combined poverty levels were compared 
(Table 8). 
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Table 8 Effect of Poverty on Test Score Gap for Various Student Subgroups 



Mean Scores 




N 


Hands-on score 


Gap 


Multiple-choice score 


Gap 


All whites 


759 


78.7 


12.3 


70.6 


15.8 


All Blacks 


368 


66.4 




54.8 




High- poverty 
Whites 


77 


73.4 


8.3 


60.8 


8.0 


Blacks 


201 






52.8 




No- poverty 
Whites 


652 


79.5 


10.1 


72.1 


13.9 


Blacks 


137 


69.4 




58.2 






All W'hites 


759 


78.7 


14.0 


70.6 


14.9 


All Hispanics 226 


64.7 




55.7 




High Poverty 
Whites 


77 


73.4 


13.3 


60.8 


7.7 


Hispanics 


107 


60,1 




53.1 




No-povert>^ 
Whites 


652 


79.5 


11.0 


72.1 


13.6 


Hispanics 


90 


68.5 




58.5 





Was Test Performance Affected More by Poverty or Reading? 

Reading was found to have a greater effect than both poverty level and race on both the 
hands-on and multiple-choice science prcx:ess skills tests. Across all racial groups, scores of no- 
poverty students w ere found to increase from low- to middle- to high-reading levels. Students at 
the no-poverty level were more affected by low-reading ability than students of high-reading level 
were affected by high-poverty (Table 9), On the hands-on test, for each race and sex, the mean 
score for the no-poverty low-reading group w^as lower than the mean score for the high-reading 
high-poverty group. Females appeared to be more affected b\' reading (abcnit two times) than 
males. 
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lable 9. Hands-on Scores for Students With Different Reading and Poverty Levels 



High Reading 



White male White female 



Black male Black female 



Hispanic male Hispanic female 



High-poverty 
No- poverty 



78.6 
8G.7 



82.7 
83.0 



70.5 
793 



73.0 
77.9 



72.8 
81.7 



79.8 
84.0 



No- Poverty 



White male White female 



Black male Black female 



Hispanic male Hispanic female 



Low- reading 72.2 70.1 
Middle-reading 77.2 77.3 
High-reading 86.7 83.0 



62.9 
71.5 
79.3 



59.5 
70.0 
77.9 



49.9 
70.6 
81.7 



60.6 
76.9 
84.0 



White male White female Black male Black female Hispanic male Hispanic female 



High-poverty ' 
high-reading 



78.6 



82.7 



70.5 73.0 



72.8 



79.8 



No-poverty/ 
low-reading 



72.2 



70.1 



62.9 59.5 



49.8 



60.() 



Difference 



6.4 



12.6 



7.6 



13.5 



22.9 



19.2 



How Does the Presence of an Elementary Science Specialist Affect Hands-On and 

Multiple Choice Scores? 

Compared lo students in a program without a science specialist, those students in a program 
with a science specialist were found to achieve significantly higner scores on the hands-on test 
(Part 111 of the ESPET), on the multiple-choice science content section of the ESPET (Part I), on 
the combined two multiple-choice sections of the ESPET (Parts I and II), on the total ESPET 
(Parts I, II, and III), and on the Iowa Tests of Basic Skills Science Subtest (Table 10, Figure 10). 
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Table 10 T- tests: Dirrerenccs in Science Test Scores With and Without a Science Snecialisi 
Science Test Mean with Mean without 



Specialist 


Specialist 


t- value 


2-tiiil prob 


ESPET hands-on 


78 


69 


9.52 


.0001 


science process skills 










liSPET multiple-choice 


G5 


64 


1.29 


.1% 


science process skills 










r^PET multiple-choice 


63 


61 


2.69 


.008 


science content 










ESPETT multiple-choice 


64 


bl 


2.84 


.005 


skills plus content 










ESPET total test 


69 


64 


5.65 


.0001 


ITBS science subtest 


49 


46 


3.41 


.001 



Note: With a science specialist, N = 577; without a Science Specialist, N = 804. 



□ without specialist with specialist 




hands- multiple multiple total ITBS 
on skills choice choice ESPET Science 
skills content 



Figure 10. ElTccl of Science Specialist on Science Test Scores of Fourth Grade Students. 



The mean score for students with a science specialist was also higher on the multiplc-clioicc 
science prcKcss skills section of the ESPET (Part II), however the dilTerencc was not signilicanl. 
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When disaggregated data is examined, it is observed that various student subgroups with a science 
specialist also score higher: males and females. Figure 11; high-, low-, and no-povcrty groups. 
Figure 12; whites, blacks and Hispanics, Figure 13; no-poverty groups and high-reading groups, 
Table 11; low- , middle-, and high-reading groups, Table 12. 



Table 11 Effect of Science Specialist on Science Test Scores for Racial Subgroups with No-Foverty 
and High-Reading Lx^vels 



Mean Scores 





HandvS-on 




Multiple-Choice 


White 


Black 


Hispanic 


White 


Black 


Hispanic 


(N) 


(N) 


(N) 


(N) 


(N) 


(N) 


all 787 


66.4 


64.7 


70.6 


54.8 


55.7 


students (759) 


(368) 


(226) 


(759) 


(:^8) 


(226) 


without Specialist 












75.5 


62.8 


60.0 


69.7 


55.8 


54.2 


(443) 


(187) 


(159) 


(443) 


(187) 


(159) 


with Specialist 












83.0 


70.2 


76.0 


71.8 


53.7 


59.2 


(316) 


(181) 


(67) 


(316) 


(181) 


(67) 


No Poverty without specialist 










76.5 


65.2 


65.3 


71.2 


57.8 


57.6 


(377) 


(64) 


(63) 


(377) 


(64) 


(63) 


No Poverty with specialist 










83.5 


73.0 


76.0 


73.2 


58.5 


60.5 


(275) 


(73) 


(27) 


(275) 


(73) 


(27) 


High Reading without specialist 








81.9 


68.4 


78.0 


81.7 


704 


75.8 


(183) 


(32) 


(28) 


(183) 


(32) 


(28) 


High Reading with specialist 










87.0 


81.0 


85.9 


81.9 


73.8 


77.4 


(152) 


(34) 


(17) 


(152) 


(34) 


(17) 


No Poverty and High Reading 


without specialist 








82.8 


70.0 


81.6 


82.2 


71.4 


78.3 


(165) 


(15) 


(14) 


(165) 


(15) 


(14) 


No Poverty and High Reading 


with specialist 








86.8 


84.5 


86.2 


82.4 


76.0 


82.1 


(136) 


(22) 




(136) 


(22) 


(8) 
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Table 12 interactive Effects of Race, Reading, Science Specialist Hands-on Tests of Science Process Skills 





Mean Scores 




Low-Reading 


Middle-Reading 


High-Reading 


White Black Hispanic 
(N) (N) (N) 


White Black Hispanic 
(N) (N) (N) 


White Black Hispanic 
(N) (N) (fsJ) 



With Specialist 

76.8 65.4 
(69) (88) 


68.8 
(24) 


81.2 
(95) 


71.3 
(59) 


76.2 
(26) 


87.0 81.0 
(335) (34) 


85.9 
(17) 


Without Specialist 

68.0 58.3 
(102) (92) 


45.9 
(74) 


73.0 
(158) 


66.3 
(63) 


69.5 
(57) 


81.9 68.4 
(183) (32) 


78.0 
(28) 



Note . ANOVA F = 2.130 p = .075. 




CH without 
specialist 



with specialist 



males 



females 



Figure 1 1. Differences in Performance on Hands-on Test of Science Prkcss Skills foi Males and 
Females With and Without a Science Specialist 
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poverty poverty poverty 



Poverty Level 



Figure 12, Hands-on Test Scores of Students of Different Poverty Levels With and Without a 
Science Specialist 




white black Hispanic 
Racial Subgroups 



n without science 
specialist 



with science 
specialist 



Figure 13. Hands-on Test Scores of Students of Different Racial Subgroups With and Without 
Science Specialist 



Alternative Forms of Assessment in Elementary Science: The Interactive Effects of Sex, Reading, 
Race, Economic Level and the Elementary Science Specialist on Hands-on and Multiple-Choice 

Assessment of Science Process Skills 



Numerous two-way and three-way interactive effects were observed between race and the 
science specialist and sex, race and the science specialist (Satumelli, 1993). For example, see Figure 
14. 



75 -r 




■ 


Hispanics 




low 




reading 


□ — 


blacks 
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Figure 14. Interactive Effects of Race, Reading, and Science Specialist on hands-on Test of 
Science Process Skills: Blacks and Hispanics with and without a Science Specialist. 



Does Alternative Assessment Make a Difference? 

Students of all reading abilities and all poverty levels in all racial subgroups appear to be able 
to demonstrate what they know and can do better on the hands-on test than on the multiple-choice 
test of science process skills (Tables 5, 6 and 7; Figures 1, 2, 8 and 9). The variation in 
performance on the two tests of science process skills resulted in a reduction in the test score gap 
between various subgroups of students including the following: no- and high-poverty Whites; 
White and Black males; no-poverty White and Hispanic females; White and Black females (Table 
4). 
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Conclusions^ 

All students performed better on the hands-on test of science process skills test, a test that 
was found to not rely heavily on reading, and that was at their cognitive level of development. The 
test score gap that has been observed to exist between students of different economic and racial or 
ethnic backgrounds was found to be reduced. On the hands-on test, economically disadvantaged 
(high poverty) students were provided the opportunity to demonstrate what they knew and could 
do and the gap between low-reading level and high-reading level students of all racial groups was 
less than on the multiple choice test. 

The hands-on test of science process skills may be considered as coming closer to measuring 
what science educators want to measure (Doran, 1990; Kanis, 1988; Cizek, 1991; Petraitis, 1991; 
Kulm and Stuessy, 1991; Maeroff, 1991; Shavelson, Baxter, and Pine, 1992), because it appears 
to be less dependent upon reading ability than the multiple-choice test of science process skills; 
because it appears to be a more developmentally approriate test (more concrete, less abstract) for 
fourth graders; and because it matches instruction, especially in hands-on science programs where 
there is a science specialist. 

The results of this study provide data needed to answer part of the question proposed by 
Jeannie Oakes (1990) who asks il* it is race or economic status that has a greater effect on science 
and math achievement. For science, it appears to be that the answer is economic status. Within 
each racial group, test scores were found to increase significantly from high- to no-po\ crty le\ cls 
and the gap between racial subgroups was less when scores of students of the same pcwc' level 
were compared than when scores of racial subgroups of combined poverty levels were compared. 
The results of this study also allow the concerns about reading, expressed by Scott-Jones and 
Clark (1986) and Tolman, Sudwecks, Baird and Tolman (1991), to be addressed. Data obtained 
leads to the conclusion that reading has an even greater effect than pcn'erty level on science 
achievement; and, as Scott-Jones and Clark suggest, a complex pattern of differential performance 
m reading exists for males and females. For low-reading ability students, females scored higher 
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than males whereas for high-reading ability students, males were found to significantly score 
higher than females. In addition, the results of this study corroborate the work of Doran and Tamir 
(1992) and Shavelson, Baxter, & Pine (1992) regarding the mode of assessment of science 
process skills. Hands-on test scores of all students were found to be higher than their multiple- 
choice test scores of science process skills. The difference was greater for some subgroups of 
student than for others. 

The results presented here (Table 10) support the studies of Beane (1985), Bredderman 
(1983), and Kyle, Shymansky and Alport (1982) who found that economically and/or 
educationally disadvantaged students in hands-on activity-based science programs performed better 
than those in text- book- based programs. From the results of this study it can be concluded that 
students in science programs with a science specialist, where the focus is teaching science through 
hands-on experiences, performed significantly better than those in programs without a science 
specialist, especially those students of low-reading and high-poverty levels. The results of this 
study also add strength to the conclusions drawn by Tamir ( 1989) and Zuzovsky and Tamir ( 1989) 
and Staver and Walberg (1986) regarding the effect of alterable school variables on certain subjects 
such as science, especially in low socio-economic schools. The findings of this study also provide 
information for those educators involved in the debate about the elementary science specialist 
(Abeil, 1990; Hounshell & Swartz, 1987). It was found that with elementary science specialists, 
science was assured of being taught, and it was apparently learned. When students were provided 
a science specialist, the opportunity to learn science increased for all subgroups of students. All 
students demonstrated that they had learned more science with a science specialist than without but 
because some subgroups exhibited a greater increase in scores with a science specialist than other 
subgroups the gaps between subgroups which differ on the basis of race, poverty, and reading is 
reduced. See Figures 14, 15 and 16 where scores of various subgroups of students are compared 
with and without a science specialist. Figure 17 clearly shows that when the three factors 
(poverty, reading, and science specialist) are accounted for, the gap between racial subgroups is 
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nearly eliminated when the hands-on scores of no-poverty, high-reading students are in a program 
with a science specialist. 
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no yes pov pov pov pov pov pov 

spec spec spec spec 

no yes no yes 



Figure 15. Effect of Poverty and Science Specialist: High- and MoPovcrty Hispanic, Black and 
White Students With and Without a Science Specialist 
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Figure 16. Effect of Reading and Science Specialist: Low- and High- Reading Hispanic, Black 
and White Students With and Without a Science Specialist 
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Figure 17. Effect of Poverty, Reading and Science Specialist on Hands- on Test Scores: High- 
Poverty Low Reading and No-Poverty High-Reading Hispanic, Black and White Students With 
and Without a Science Specialist 

Table 13 Student Sample Available for Study 



Student Subgroup 


N 


Student Subgroup 


N 


Total sample 


1381 


VVTiite females 




Males 


677 


liigh-poverty 


36 


Fximales 


704 


Low-fxjverly 


18 


Whites 


759 


No-poverty 


322 


Blacks 


368 


Black males 




Hispanics 


226 


High-poverty 


87 


OLKers 


28 


Low-pt)verty 


13 


high-pi>verly 


388 


No -poverty 


74 


Low-poverty 


94 


Black females 




No- poverty' 


899 


High-povert\' 


114 


White males 


383 


Low-poverty 


17 


White females 


376 


No-povefty 


63 


Black males 


174 


Hispanic m.ales 




Black females 


194 


High-poverty 


46 


Hispanic males 


103 


Low-poverty 


18 


Hispanic females 


123 


No-poverty 


39 


Other males 


16 


Hispanic females 




Other females 


12 


High-pi>verty 


62 


White males 




Low-poverty 


11 


High-f>overly 


41 


No-poverty 


50 


U)W-pi>verty 


12 






No-poverty 


330 






Other males 




Other females 




1 Hgh-poverty 


1 


High-pi>verty 


0 


Low- poverty 


3 


Low-poverty 


2 


No-poverty 


12 


No-poverty 


10 


Science specialist 








With 


577 






Without 


804 
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Educational Importance. 

Jeannie Oakes (1990) asked if it is race or economic status that has a greater effect on 
science and math achievement. Based upon the results of this study, it appears that for science the 
answer is economic status. Within each racial group, test scores were found to increase 
significantly from high-poverty to no-poverty levels. This study shows that when economically 
disadvantaged (high-poverty) students are tested in alternative ways, they are better able to 
demonstrate what they know and can do. 

The results of this study also a.lovv concerns about reading as expressed by Scott-Jones and 
Clark (1986) and Tolman, Sudweeks, Baird, & Tolman (1991) to be addressed. On the hands-on 
test, students with low reading levels were apparently less handicapped by their inability to read 
and therefore performed better on the hands-on performance test than they did on the multiple- 
choice test which relies more upon reading. Data obtained lead to the conclusion that reading has an 
even greater effect than poverty level on science achievement; and, as Scott-Jones and Clark 
suggest, a complex pattern of differential perlbrmance in reading was found to exist for males and 
females. For low-reading ability students, females scored higher than males, whereas for high- 
reading ability students, males were found to score higher than females. 

In addition, the results of this study corroborate the work of Doran and Tamir (1992) 
regarding the mcxie of assessment of science process skills. Hands-on test scores of all students 
were found to be higher than their multiple-choice test scores of science process skills. The 
difference was greater for some subgroups of students than for others and because of this, the gap 
between certain subgroups of students was greatly reduced. 

The results of this study provide additional evidence to support the studies of Bcanc (1985), 
Brcddcrman (1983), and Kyle, Shymansky, & Alport (1982) who found that economically and or 
educationally disadvantaged students in hands-on activity-based science programs performed better 
than those in tcxtbook-bascd programs. From the results of this study, it can be concluded that 
students in science programs where the fcx:us is teaching science through hands-on aclivity-bascd 
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experiences (with a science specialist), performed significantly better than those in programs 
without a hands-on program (without a science specialist), especially those students of low-reading 
and high-poverty levels. 

The results of this study also add strength to the conclusions drawn by Zuzovsky & Tamir 
( 1989) and Slaver & Walbcrg ( 1986) regarding the effects of alterable school variables on certain 
subjects such as science, especially in lov/-socio-economic schools. A science specialist is an 
alterable school variable. The science specialists in the schools in this study were not teachers with 
special science degrees or certification. They were elementary teachers who chose and were 
selected to teach science in their schools. They wanted to teach science. Therefore, it can be 
inferred that not only are we assured that science was taught regularly and frequently, but that they 
were probably enthusiastic about teaching it and, in turn, these teachers most probably conveyed 
this positive attitude about science to their students. The final result is that someone who wanted to 
teach science was accountable for teaching it and did so regularly. Hence, science was taught 
using a hands-on approach and when it was taught more science was apparently learned by all 
students. 

Based upon the results of this study, it is clear that all students can learn science provided 
that two conditions are met: ( 1) students must be provided with appropriate instruction so that they 
have the opportunity to learn science (this can be assured by providing a science specialist); and (2) 
students must be able to demonstrate what they know and can do (this condition can be met by 
providing hands-on, performance-based assessment). 
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